Projection system, device and method for the output of calibration projection scenes

ABSTRACT

A projection system includes projection units configured to project an image on a projection body; a preparing unit configured to prepare calibration-use images; an extraction unit configured to extract, from each of the calibration-use images, at least grating points indicating a distortion in a projected image of one of the projection units and alignment points of the projected image of the one of the projection units or another one of the projection units; a conversion unit configured to convert, onto a common coordinate system, the grating points of the projected images of the projection units extracted from the calibration-use images, based on alignment points common to the calibration-use images; and a geometric correction coefficient calculation unit configured to calculate a geometric correction coefficient for providing a projection image to be projected from the projection units, based on the grating points on the common coordinate system.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a projection system, an imageprocessing device, and a projection method. More specifically, thepresent invention relates to a projection system for projecting an imageon a projection body by a plurality of projection units, an imageprocessing device constituting the projection system, and a projectionmethod executed by the projection system.

2. Description of the Related Art

Conventionally, the multi-projection technology is garnering attention,in which projected images from a plurality of projectors are arrangedwith areas overlapping with each other, and a single high-resolutionimage is projected on a screen.

A known example of the above multi-projection technology is described inJapanese Patent No. 3908255 (Patent Document 1). In the image projectionsystem of Patent Document 1, a reference image is projected onto ascreen from the respective projectors. The reference image includes fouror more feature points whose coordinate positions are known in advance.The reference image is a known image such as a grating pattern in whichbright spots or crosses are arranged with fixed intervals. Then, thepositions of the feature points in the reference image, which is takenby (imaged by) a digital camera, are detected. Based on the detectedpositions of four or more feature points of each projector, theprojection image of each projector is deformed, and the overlappingareas are detected and a blending process is performed. The projectedimages which have been deformed and which have undergone the blendingprocess, are projected from the plurality of projectors, and arranged onthe screen with areas overlapping each other, to form a singlehigh-resolution image.

When performing the multi-projection described above, in order to alignthe projection images and match the scale of the projection images, itis necessary to sequentially or simultaneously project reference imagesfrom the projectors, and take images of the reference images. However,by the method of the conventional technology in which the referenceimages are sequentially projected from the projectors, and the imagesare taken a plurality of times, the camera needs to be fixed on atripod, and the image needs to be taken such that the projection rangesof all projectors are included in the angular field. Therefore, thistechnology has been unsatisfactory in that equipment such as a tripod isnecessary, which reduces the convenience. Furthermore, if the number ofprojectors increases, there have been cases where it is difficult totake an image to include the projection ranges of all projectors in theangular field at once. For example, when multi-projection is performedon the wall of a hallway, due to restrictions such as the width of thehallway, it is difficult to secure a sufficient distance for taking animage by including the projection ranges of all projectors in theangular field.

Meanwhile, by a method of the conventional technology of simultaneouslyprojecting reference images from projectors and taking an image of thereference images, the structure patterns of bright spots and crosses inthe reference images from projectors that are simultaneously projected,overlap each other, and the attribution of the patterns need to bedetermined in image processing. In this case, when the patterns of thedifferent projectors adhere to each other, it is difficult to separatethe patterns and determine the attribution of the patterns. Therefore,the conventional technology has been unsatisfactory.

Furthermore, Japanese Patent No. 3497805 (Patent Document 2) discloses atechnology of performing split imaging, by which the image is takenwithout including the projection ranges of all projectors in the angularfield described above. However, in order to combine the images taken bysplit imaging described in Patent Document 2, it is necessary toaccurately control the position and the orientation of the camera whenperforming split imaging, and an exclusive-use position control deviceis required for this camera control. Therefore, the conventionaltechnology of split imaging described in Patent Document 2 has beenunsatisfactory in terms of the ease in calibration and cost.Furthermore, the problem of the structure patterns overlapping eachother is not addressed in Patent Document 2.

Japanese Laid-Open Patent Publication No. 2012-47849 (Patent Document 3)is known as a technology of stack projection, in which when a pluralityof projectors project images on a projection body to overlap each other,the structure patterns are simultaneously projected in an overlappingmanner, an image is taken of the structure patterns, and the structurepatterns are later separated. The conventional technology of PatentDocument 3 discloses a method in which patterns whose wavelength regionsof R, G, B have been changed for each projector are projected, andpatterns whose polarization properties have been changed are projected,and the superposed patterns are separated later based on the wavelengthsand the polarization properties. However, by the method of projectingpatterns whose wavelength regions have been changed, the wavelengthregions of R, G, B, of a projector and the wavelength regions of R, G,B, of a camera usually do not match, and therefore it has not been easyto separate the patterns into separate color signals by using a typicalcamera. By the method of projecting patterns whose polarizationproperties have been changed, an exclusive-use imaging device isnecessary, which leads to increased cost.

Furthermore, Japanese Laid-Open Patent Publication No. 2011-182076(Patent Document 4) discloses a method of simultaneously projecting,with a plurality of projectors, a plurality of types of patterns whosephases are shifted from each other, by devising a way to position thepatterns so as not to overlap each other, and taking an image of theprojected patterns. However, in order to ensure precision in patternextraction, it is necessary to project patterns having a sufficientsize. Meanwhile, it is necessary to reduce the pattern intervals inorder to increase the spatial density of patterns for the purpose ofincreasing the precision in alignment. Furthermore, in an ultra-shortfocus projector that has recently become available, images are projectedfrom a close distance to the screen, and therefore the projected imagewill easily become distorted in a non-linear manner, due to factorsrelevant to focusing or slight setting conditions, or slightirregularities on the screen. For these reasons, there has been a limitin the method of simultaneously projecting patterns with a plurality ofprojectors while avoiding the overlapping of the patterns, and taking animage of the projected patterns.

Patent Document 1: Japanese Patent No. 3908255

Patent Document 2: Japanese Patent No. 3497805

Patent Document 3: Japanese Laid-Open Patent Publication No. 2012-47849

Patent Document 4: Japanese Laid-Open Patent Publication No. 2011-182076

SUMMARY OF THE INVENTION

The present invention provides a projection system, an image processingdevice, and a projection method, in which one or more of theabove-described disadvantages are eliminated.

According to an aspect of the present invention, there is provided aprojection system including a plurality of projection units configuredto project an image on a projection body; a taken image preparation unitconfigured to prepare a plurality of calibration-use images; anextraction unit configured to extract, from each of the plurality ofcalibration-use images, at least grating points indicating a distortionin a projected image of one of the plurality of projection units andalignment points of the projected image of the one of the plurality ofprojection units or a projected image of another one of the plurality ofprojection units; a conversion unit configured to convert, onto a commoncoordinate system, the grating points of the projected images of theplurality of projection units extracted from the plurality ofcalibration-use images by the extraction unit, based on alignment pointscommon to the plurality of calibration-use images; and a geometriccorrection coefficient calculation unit configured to calculate ageometric correction coefficient for providing a projection image to beprojected from the plurality of projection units, based on the gratingpoints on the common coordinate system.

According to an aspect of the present invention, there is provided animage processing device for performing projection with the use ofplurality of projection units, the image processing device including ataken image preparation unit configured to prepare a plurality ofcalibration-use images; an extraction unit configured to extract, fromeach of the plurality of calibration-use images, at least grating pointsindicating a distortion in a projected image of one of the plurality ofprojection units and alignment points of the projected image of the oneof the plurality of projection units or a projected image of another oneof the plurality of projection units; a conversion unit configured toconvert, onto a common coordinate system, the grating points of theprojected images of the plurality of projection units extracted from theplurality of calibration-use images by the extraction unit, based onalignment points common to the plurality of calibration-use images; anda geometric correction coefficient calculation unit configured tocalculate a geometric correction coefficient for providing a projectionimage to be projected from the plurality of projection units, based onthe grating points on the common coordinate system.

According to an aspect of the present invention, there is provided aprojection method of projecting an image on a projection body by aplurality of projection units, the projection method includingpreparing, by a computer, a plurality of calibration-use images;extracting, by the computer, from each of the plurality ofcalibration-use images, at least grating points indicating a distortionin a projected image of one of the plurality of projection units andalignment points of the projected image of the one of the plurality ofprojection units or a projected image of another one of the plurality ofprojection units; converting, by the computer, onto a common coordinatesystem, the grating points of the projected images of the plurality ofprojection units extracted from the plurality of calibration-use imagesat the extracting, based on alignment points common to the plurality ofcalibration-use images; and calculating, by the computer, a geometriccorrection coefficient for providing a projection image to be projectedfrom the plurality of projection units, based on the grating points onthe common coordinate system converted at the converting.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects, features and advantages of the present invention willbecome more apparent from the following detailed description when readin conjunction with the accompanying drawings, in which:

FIG. 1 is a schematic diagram illustrating the overall configuration ofa projection system according to an embodiment;

FIG. 2 is a functional block diagram of the projection system accordingto an embodiment;

FIGS. 3A and 3B illustrate examples of two types of calibration imagesused in the projection system according to the embodiment;

FIG. 4 is for describing how a calibration scene selection unitsequentially selects calibration projection scenes, and a method oftaking the calibration projection scenes, in a first mode;

FIG. 5 is for describing how the calibration scene selection unitsequentially selects calibration projection scenes, and the method oftaking the calibration projection scenes, in a second mode;

FIG. 6 is a flowchart indicating the overall flow of the calculationprocess of calculating various correction coefficients, and a correctionprocess based on the correction coefficients, according to theembodiment;

FIG. 7 is a flowchart indicating a process of calculating a geometriccorrection coefficient executed by a correction coefficient calculationunit according to the embodiment;

FIG. 8 is for describing three calibration-use images prepared by takingimages of calibration projection scenes, and a projection conversioncoefficient that is calculated among these taken images, in the firstmode;

FIG. 9 is for describing two calibration-use images prepared by takingimages of calibration projection scenes, and a projection conversioncoefficient that is calculated among these taken images, in the secondmode;

FIG. 10 schematically illustrates an assembly of grating pointcoordinates of projectors combined on a common coordinate system;

FIGS. 11A and 11B illustrate a method of calculating outer peripherycoordinates of a projection possible area according to linearextrapolation by using grating point coordinates that have beencombined;

FIG. 12 is for describing projection possible areas of three projectorson the common coordinate system, a projection target area aftercorrection, and a projection content image;

FIG. 13 is for describing the association of coordinates in theprojector memory and coordinates on an equal-magnification content imagecorresponding to positions on a projection content image;

FIG. 14 is a flowchart of a process of calculating a blendingcoefficient executed by a correction coefficient calculation unit,according to the embodiment;

FIG. 15 is for describing the association of blending coefficients withrespect to coordinates in the projector memory;

FIG. 16 illustrates a graph of input output properties of a projector;

FIG. 17A illustrates a data structure of a geometric correctioncoefficient;

FIG. 17B illustrates a data structure of a blending coefficient;

FIG. 18 describes a correction process based on a correctioncoefficient, executed by a correction processing unit according to theembodiment;

FIG. 19A illustrates examples of a first calibration image Cij, a secondcalibration image Aij, and a third calibration image Cij+Aij;

FIG. 19B illustrates the overlapping of grating patterns;

FIG. 20 illustrates an example where three projected images areconnected in the horizontal direction, and a method of taking an imageof these projected images;

FIG. 21 illustrates an example where three projected images areconnected in the vertical direction, and a method of taking an image ofthese projected images;

FIG. 22 illustrates a calibration projection scene of projected imagesin three lines and three rows, and a method of taking images byprioritizing the frequency of taking images;

FIG. 23 illustrates a calibration projection scene of projected imagesin three lines and three rows, and a method of taking an image byprioritizing the angular field; and

FIG. 24 illustrates a hardware configuration of a general-purposecomputer according to the embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A description is given, with reference to the accompanying drawings;however, the present invention is not limited to the embodimentsdescribed below. Note that in the embodiments described below, anexample of a projection system is described by a projection system 100including a plurality of projectors which are projection units, a singlecamera which is an imaging unit, and an image processing device whichperforms overall control.

Overall Configuration

FIG. 1 is a schematic diagram illustrating the overall configuration ofthe projection system 100 according to the present embodiment. Theprojection system 100 illustrated in FIG. 1 includes an image processingdevice 110 for performing the overall control of the system, a pluralityof projectors 150, and a camera 160. Note that in the embodimentdescribed below, the projection system 100 has a configurationcorresponding to so called large-sized screen multi-projection, in whichthe projected images of three projectors 150 a through 150 c arecombined on the projection surface, and the combined image is projectedin an area that is larger than the case of using a single projector.

The image processing device 110 is typically a general-purpose computersuch as a personal as a work station. Note that the image processingdevice 110 is not limited to a general-purpose computer; the imageprocessing device 110 may be implemented as an exclusive-use computer,or may be incorporated in one of the projectors 150.

Each of the projectors 150 is a projection device that uses, forexample, a liquid crystal method, a CRT (Cathode Ray Tube) method, a DLP(Digital Light Processing) method, or a LCOS (Liquid Crystal On Silicon)method.

The camera 160 is an imaging device including an imaging sensor such asa CMOS (Complementary Metal Oxide Semiconductor) or a CCD (ChargeCoupled Device), and an imaging optical system such as a lens forimaging an image on a light receiving area of the image sensor. Thecamera 160 may be an exclusive-use device such as a web camera, adigital still camera, and a digital video camera, or a deviceincorporated in a general-purpose device such as a smartphone terminaland a tablet terminal.

In the projection system 100, a screen 102 is set for providing aprojection surface. The projectors 150 are arranged for projectingimages on the screen 102, such that the positions of projection centersof the projectors 150 are shifted from each other. The image processingdevice 110 generates a plurality of projection images to be projected bythe plurality of projectors 150 a through 150 c, and outputs therespective projection images to the corresponding projectors 150. Theprojectors 150 project, on the screen 102 that is a projection body, theprojection images input to the respective projectors 150 from the imageprocessing device 110. As illustrated in FIG. 1, on the screen 102, aplurality of projected images 104 a through 104 c are projected from theplurality of projectors 150 a through 150 c, respectively. The pluralityof projected images 104 a through 104 c from the plurality of projectors150 a through 150 c are superposed on the projection surface, andcombined into a single projected image 106.

During a projection mode, the projection system 100 uses the pluralityof projectors 150 a through 150 c to project a single projected image106 as described above; however, before the projection mode, acalibration process is usually performed. The camera 160 illustrated inFIG. 1 is used for this calibration process. During the calibrationmode, the image processing device 110 outputs calibration images to theplurality of projectors 150, and causes the projectors 150 to project acalibration-use projected image (projected image used for calibration)on the screen 102. Then, the viewpoint and the vision of the camera 106are set, such that projected images 104 projected by the predeterminedprojectors 150 fit inside the angular field of the camera 160. Thecamera 160 takes images (performs imaging) of the calibration-useprojected images for a plural number of times in order to performcalibration.

The taken image taken by the camera 160 (hereinafter, a taken image inwhich a calibration-use projected image is included, is referred to as a“calibration-use image”) is sent to the image processing device 110 bywireless connection such as wireless LAN (Local Area Network), Bluetooth(registered trademark), wireless USB (Universal Serial Bus), or wiredconnection such as wired USB and wired LAN. Alternatively, thecalibration-use image taken by the camera 160 is scanned by the imageprocessing device 110 via a removable medium such as a SD card(registered trademark) or compact flash (registered trademark).

The image processing device 110 uses the plurality of inputcalibration-use images to calculate various correction coefficients foraligning the positions, matching the scale, correcting the distortion,and adjusting the brightness of the overlapping areas, with respect tothe projected images of the plurality of projectors 150 a through 150 c.During the projection mode, the image processing device 110 generates aprojection image that has been corrected in order to be projected by theprojectors 150 a through 150 c, based on the respective correctioncoefficients that have been calculated. In the following, with referenceto FIGS. 2 through 5, a description is given of an overview of acalculation process of calculating the respective correctioncoefficients and a correction process based on the correctioncoefficients.

Overall Functional Configuration

FIG. 2 is a functional block diagram of the projection system 100according to the present embodiment. The projection system 100illustrated in FIG. 2 includes a plurality of functional blocks thatoperate on the image processing device 110. The image processing device110 includes a content storage unit 112, correction processing units 114a through 114 c of the respective projectors, projected image outputunits 116 a through 116 c of the respective projectors, and switchingunits 122 a through 122 c of the respective projectors. The imageprocessing device 110 further includes a calibration image storage unit118, a calibration scene selection unit 120, a calibration-use imageinput unit 124, and a correction coefficient calculation unit 130.

The content storage unit 112 stores a file of a content image that isthe target to be projected as the single projected image 106. Thecontent storage unit 112 is used as a storage area of a HDD (Hard DiskDrive), a SSD (Solid State Drive), and a detachably attached removablemedium. Note that the content image that is the projection target may begiven as a display screen when a word processor or an application of apresentation executes a file, or may be given as a still image, or maybe given as a frame of an arbitrary timing in a video file. Furthermore,the content image need not be given as file; the content image may begiven as a screen generated as the image processing device 110 executesthe operating system, or as a projected image input to the imageprocessing device 110 from outside. In the following, as a matter ofconvenience, a description is given of an example where the contentimage is given as a still image.

The correction processing units 114 a through 114 c are provided tocorrespond to the projectors 150 a through 150 c included in theprojection system 100, respectively. Each of the correction processingunits 114 reads a content image from the content storage unit 112,performs a correction process on the content image, and generates aprojection image for the corresponding projector. Note that details ofthe processes executed by the correction processing units 114 a through114 c are described below.

The projected image output units 116 a through 116 c are provided tocorresponding to the projectors 150 a through 150 c included in theprojection system 100, respectively. Each of the projected image outputunits 116 includes a display output connected to the correspondingprojector 150, and outputs, to the connected projector 150, a projectedimage of the input image selected at the switching unit 122.

The switching units 122 a through 122 c switch the flow of the imageaccording to the operation mode of the projection system 100. During theprojection mode of projecting the content image, the switching unit 122switches the input side to the output of the correction processing unit114. In accordance with this switching operation, during the projectionmode, the projected image output unit 116 outputs a projected image ofthe processing result based on the content image according to thecorresponding correction processing unit 114. Meanwhile, during thecalibration mode, the switching unit 122 switches the input side to theoutput of the calibration scene selection unit 120 described below. Inaccordance with this switching operation, during the calibration mode,each of the projected image output units 116 outputs a projected imageof the calibration image selected and output by the calibration sceneselection unit 120.

The calibration image storage unit 118 stores a calibration image to beprojected from the projector 150 during the calibration mode. Thecalibration image storage unit 118 is used as a storage area of a HDD, aSSD, and a detachably attached removable medium. The calibration imageis typically provided as a still image that is prepared in advance.

The calibration image may include both of or one of a grating patternthat defines the grating points (points on the coordinate system of eachof the calibration-use images including a grating pattern) in theprojected image, or an aligning pattern that defines the alignmentpoints in the projected image. FIGS. 3A and 3B illustrate examples oftwo types of calibration images used in the projection system 100according to the present embodiment. FIG. 3A illustrates an example of afirst calibration image 200 including both an alignment pattern 202 anda grating pattern 206. FIG. 3A illustrates an example of a secondcalibration image 210 including only an alignment pattern 212.

The grating pattern 206 is for defining coordinates in the projectormemory, and includes patterns in which arbitrary figure elements arearranged by a predetermined rule. By taking an image of the gratingpattern 206 projected on the screen 102, it is possible to detecttrapezoidal distortions and local distortions in the projected image. Inthe first calibration image 200 illustrated in FIG. 3A, the gratingpattern 206 divides the entire projection area of the projector 150 intoten blocks in the horizontal direction and in seven blocks in thevertical direction, and in the center 8×5 blocks among these blocks,solid white circles 204 are arranged in a grating pattern on a blackbackground.

However, the grating pattern 206 is not particularly limited; variouskinds of patterns may be used, such as polka-dots in which circleshaving a contrast with respect to the background as illustrated in FIG.3A are arranged two-dimensionally, a dot pattern in which dots having acontrast with respect to the background are arranged two-dimensionally,a checkered pattern in which squares of two colors having a contrastwith each other are alternately arranged in the horizontal and verticaldirections, and a grating pattern in which lines having a contrast withrespect to the background are arranged two dimensionally. The method ofdividing the entire projection area of the projector 150 is not limitedto the above embodiment; the number by which the area is divided and thedivision method of the area may be determined according to the requiredprecision and the performance of the image processing device 110.

The alignment patterns 202, 212 are for defining the reference positions(alignment points) of the projected images among the taken images, andare patterns in which a plurality of arbitrary figure elements arearranged at predetermined positions. By taking a plurality of imagesincluding the common alignment patterns 202, 212 projected on the screen102, it is possible to perform alignment among the plurality of takenimages.

In the first calibration image 200 including both the alignment patternand the grating pattern, preferably, as illustrated in FIG. 3A, thealignment pattern 202 is arranged at a position around the area wherethe grating pattern 206 is arranged, as illustrated in FIG. 3A. Also inthe second calibration image 210 including only the alignment pattern,as illustrated in FIG. 3B, the alignment pattern 212 is arranged at thesame position as the calibration image of FIG. 3A (position around areaof grating pattern if grating pattern is included).

In the first calibration image 200 illustrated in FIG. 3A, in thealignment pattern 202, rectangular markers 202LT, 202RT, 202LB, and202RB are arranged at the four corners of the outer periphery of the10×7 blocks in the enter projection image area of the projector 150.Also in the second calibration image 210 of FIG. 3B, in the alignmentpattern 212, rectangular markers 212LT, 212RT, 212LB, and 212RB arearranged at the four corners of the outer periphery of the 10×7 blocks.

However, the alignment patterns 202, 212 are not particularly limited.The shapes of the markers in the alignment patterns 202, 212 may be anarbitrary figure element such as a circle, and the number of markers maybe any number as long as there are at least four points. Note that byincreasing the number of markers used for alignment, the alignmentprecision can be improved.

Referring back to FIG. 2, in the calibration process according to thepresent embodiment, images are taken over a plurality of times, of thegrating pattern for detecting geometric distortions in the projectedimage of the projector 150, and the results of the plurality of takenimages are combined by the alignment pattern. The calibration sceneselection unit 120 reads the respective calibration images from thecalibration image storage unit 118, selects appropriate calibrationimages, and outputs the selected calibration to the plurality ofprojectors 150 a through 150 c. Here, the calibration scene selectionunit 120 has recognized the positional relationships between theprojected images of the plurality of projectors 150, and the calibrationimage is selected according to the respective stages of the calibrationprocess, such that sufficient calibration results of the projectors 150can be obtained overall. A scene of each stage of the calibrationprocess including a calibration image to be projected by at least one ofthe projectors 150, is referred to as a calibration projection scene.

According to the calibration scene selection unit 120, the respectiveprojectors 150 are caused to project calibration images according to thecalibration projection scene. At this time, the user uses the camera 160to take an image of each calibration projection scene, such that theprojected calibration-use projected images fit in the angular field. Thecalibration-use image input unit 124 receives input of the taken imagesfrom the camera 160 via wireless connection, wired, connection, or aremovable medium, and prepares a plurality of calibration-use images forthe calibration process. Note that at least in one calibrationprojection scene, the user is required to take an image by directlyfacing the screen. Typically, a water level is used to take a firstimage by directly facing the screen. In this case, when taking thesecond image and onward, there is no need for the user to directly facethe screen. The calibration-use image taken by directly facing thescreen 102 is used as a reference for combining the results.

In the present embodiment where three projectors 150 a through 150 c areused as illustrated in FIG. 1, it is possible to take a calibration-useimage by forming calibration projection scenes of two modes asillustrated in FIGS. 4 and 5. FIG. 4 is for describing how thecalibration scene selection unit 120 sequentially selects thecalibration projection scenes, and the method of taking the calibrationprojection scenes, in the first mode. FIG. 5 is for describing how thecalibration scene selection unit 120 sequentially selects thecalibration projection scenes, and the method of taking the calibrationprojection scenes, in the second mode.

In the first mode, as illustrated in FIG. 4 three calibration projectionscenes are prepared, corresponding to images taken three times. In thefirst calibration projection scene, the first projector 150 a projectsthe first calibration image 200 illustrated in FIG. 3A, and the secondprojector 150 b projects the second calibration image 210 illustrated inFIG. 3B. The third projector 150 c does not project anything. In thefirst calibration projection scene, the camera 160 is used to takeimages such that projected images 230 a, 230 b of the first and secondprojectors 150 a, 150 b fit in the vision.

In the second calibration projection scene, the first projector 150 adoes not project anything, the second projector 150 b projects the firstcalibration image 200 illustrated in FIG. 3A, and the third projector150 c projects the second calibration image 210 illustrated in FIG. 3B.In the second calibration projection scene, the camera 160 is used totake images such that projected images 232 b, 232 c of the second andthird projectors 150 b, 150 c fit in the vision. In the last thirdcalibration projection scene, the third projector 150 c projects thefirst calibration image 200 illustrated in FIG. 3A. The first and secondprojectors 150 a, 150 b do not project anything. In the thirdcalibration projection scene, an image is taken such that the projectedimage 234 c of the third projector 150 c fits in the vision.

Note that in FIG. 4, three calibration projection scenes are preparedcorresponding to the three projectors 150 a through 150 c for arrangingthree projected images in the horizontal direction. However, bygeneralizing with respect to a N (N≧2) number of projectors 150, an Nnumber of calibration projection scenes are to be constituted asfollows. That is to say, the n (1≦n≦N−1)th calibration projection sceneis prepared, such that one of the projectors arranged adjacent to eachother (nth projector) projects a calibration image including analignment pattern as illustrated in FIG. 3A (the first projector doesnot necessarily need to project an alignment pattern) and at least agrating pattern, and the other one of the adjacent projectors (n+1thprojector) projects a calibration image including only an alignmentpattern as illustrated in FIG. 3B. The last Nth calibration projectionscene is prepared such that the last projector (Nth projector) projectsa calibration image including both a grating pattern and an alignmentpattern as illustrated in FIG. 3A. In the two calibration-use images,the results are combined by using the alignment pattern projected by acommon projector 150.

In the first mode described above, the range that is fit in one imagetaken by the camera 160 is a projection range corresponding to twoprojectors at maximum. Thus, even if the number of projectors Nincreases, the restriction (of the position of the camera 160) in thedepth direction with respect to the screen 102 is alleviated, andtherefore the screen may be arranged in various ways. Note that the sameis applicable even if the projected images are arranged in a verticaldirection, or if the projected images are arranged two-dimensionally ina unicursal manner.

In the second mode, as illustrated in FIG. 5, two calibration projectionscenes are prepared, corresponding to two taken images. In the firstcalibration projection scene, the first projector 150 a and the thirdprojector 150 c project the first calibration image 200 illustrated inFIG. 3A, and the second projector 150 b projects the second calibrationimage 210 illustrated in FIG. 3B which only includes the alignmentpattern. In the second calibration projection scene, the secondprojector 150 b projects the first calibration image 200 illustrated inFIG. 3A. The first and third projectors 150 a, 150 c do not projectanything. In the first calibration projection scene, an image is takenby the camera 160 such that the projected images 240 a through 240 c ofthe projectors 150 a through 150 c fit in the vision of the camera 160.In the second calibration projection scene, an image is taken by thecamera 160 such that a projected image 242 b of the second projector 150b fits in the vision of the camera 160.

Note that in FIG. 5, two calibration projection scenes are prepared,corresponding to the three projectors 150 a through 150 c for arrangingthree projected images in the horizontal direction. By generalizing withrespect to an N(N≧3) number of projectors 150, the two calibrationprojection scenes are to be constituted as follows. That is to say, inthe first calibration projection scene, in the arrangement of projectedimages, a first group of projectors (for example, the projectors in theodd number order) alternately selected from the plurality of projectors150 are constituted to project a first calibration image including atleast a grating pattern. In the second calibration projection scene, inthe arrangement of projected images, a second group of projectors (forexample, the projectors in the even number order) respectively arrangedadjacent to the projectors of the first group are constituted to projectgrating patterns. Furthermore, the first calibration projection sceneand the second calibration projection scene are constituted such that atleast one common projector projects an alignment pattern.

More preferably, in the first calibration projection scene, theprojectors of the second group (for example, the projectors in the evennumber order) are constituted to project second calibration imagesincluding only the alignment pattern. In the second calibrationprojection scene, the projectors of the second group are constituted toproject first calibration images including both a grating pattern and analignment pattern.

In the second mode described above, although the range that is fit inone image taken by the camera 160 is large, only two images need to betaken. Therefore, when the problem in the restriction (of the positionof the camera 160) in the depth direction with respect to the screen 102can be avoided, the calibration operation can be simplified. Note thatthe same applies to the case of arranging the projected images in thevertical direction. Furthermore, in the above description, an image ofthe entire area is taken one time by each of the first group and thesecond group. However, in another embodiment, in order to reduce thenecessary angular field, in both the first and second groups, the areamay be divided and images may be taken over a plurality of times, andthe images may be combined according to an alignment pattern that iscommon in the group.

Referring back to FIG. 2, the correction coefficient calculation unit130 reads a plurality of calibration-use images prepared by thecalibration-use image input unit 124, and calculates various correctioncoefficients to be set in the correction processing units 114 a through114 c. It is assumed that the respective calibration-use images and thecalibration scenes are given, in association with each other, to thecalibration-use image input unit 124. More specifically, the correctioncoefficient calculation unit 130 includes a feature point extractionunit 132, a grating point conversion unit 134, a geometric correctioncoefficient calculation unit 136 for calculating a geometric correctioncoefficient, and a blending coefficient calculation unit 138 forcalculating a blending correction coefficient.

The feature point extraction unit 132 extracts feature points from eachof the plurality of calibration-use images that have been prepared. Thefeature points that are extracted may include grating points indicatingthe distortion of the projected image corresponding to the grating pointpattern of the calibration image, and alignment points that are used asreferences of alignment between the calibration-use images correspondingto the alignment pattern of the calibration image.

The grating points of the each of the projectors and the alignmentpoints of the taken images extracted by the feature point extractionunit 132, are passed to the grating point conversion unit 134. Thegrating point conversion unit 134 converts, onto a common coordinatesystem, the grating points of the projection images of the projectors150 extracted from the plurality of calibration-use images by thefeature point extraction unit 132 (at this stage, the grating points arepoints on the coordinate system of each of the calibration-use images),and combines these grating points, based on common alignment pointsamong the calibration-use images. In the described embodiment, thecommon coordinate system is a coordinate system of the firstcalibration-use image taken by directly facing the screen 102.

The grating points of the respective projectors converted onto thecommon coordinate system by the grating point conversion unit 134, arepassed to the geometric correction coefficient calculation unit 136. Thegeometric correction coefficient calculation unit 136 calculates, basedon the grating points on the common coordinate system, the geometriccorrection coefficients of the respective projectors to which projectionimages to be projected from the plurality of projectors 150 are given.The geometric correction coefficient is a correction coefficient inwhich geometric corrections are incorporated, such as alignment, scalematching, and distortion correction.

The blending coefficient calculation unit 138 detects an overlappingarea with respect to each of the plurality of projectors 150. Theoverlapping area is the area where a projected image of a targetprojector (for example, 150 a) and a projected image of each projector(for example, 150 b) adjacent to the target projector, overlap eachother. The blending coefficient calculation unit 138 calculates theblending coefficient for adjusting the overlapping of these projectedimages, based on the detection result of the overlapping area. Accordingto the blending coefficients of each of the projectors, the images aresmoothly combined at parts where the projected images of the pluralityof projectors 150 overlap each other on the screen 102.

Each of the correction processing units 114 generates a projection imagefor each projector from the content image, based on various correctioncoefficients calculated by the geometric correction coefficientcalculation unit 136 and the blending coefficient calculation unit 138.More specifically, the correction processing unit 114 first generates anintermediate image for each projector from the content image, based onthe geometric correction coefficient calculated by the geometriccorrection coefficient calculation unit 136. The intermediate image isformed by deforming the image handled by the projector of the contentimage, in an inverse manner with respect to the detected geometricdistortion. Next, the correction processing unit 114 multiplies theintermediate image by the blending coefficient calculated by theblending coefficient calculation unit 138, and calculates the finalprojection image for each projector. During the projection mode, theswitching unit 122 selects the projection image calculated by thecorrection processing unit 114, and the selected projection image isoutput to the projector 150.

Note that in the embodiment illustrated in FIG. 2, the respectivefunctional units 112 through 138 are realized in a single imageprocessing device 110; however, an embodiment of the projection system100 is not limited to that of FIG. 2. In another embodiment, in order toreduce the load of the image processing apparatus caused by an increasein the number of projectors, the correction processing units 114 athrough 114 c may be realized in the projectors 150 a through 150 c,respectively. In another embodiment, the respective functional units 112through 138 may be implemented by being distributed across a pluralityof image processing devices, or all of the functional units 112 through138 may be implemented in one of the projectors 150, or the functionsmay be implemented as a single device including the functions of theimage processing device 110 and the functions of the plurality ofprojectors. Furthermore, in another embodiment, the functions of thecorrection coefficient calculation unit 130 may be implemented as aserver providing services via a network.

Overall Process Flow

In the following, with reference to FIG. 6, a description is given ofthe overall flow of the calculation process of calculating variouscorrection coefficients, and the correction process based on thecorrection coefficients. FIG. 6 is a flowchart indicating the overallflow of the calculation process of calculating various correctioncoefficients, and the correction process based on the correctioncoefficients. The process of FIG. 6 is started from step S100 inresponse to an instruction from the user to start a calibration process.Note that in FIG. 6, the first mode indicated by steps S101 through S104and the second mode indicated by steps S105 through S108 are bothcollectively illustrated.

In the first mode illustrated in FIG. 4, in step S101, the imageprocessing device 110 causes the first projector 150 a to output a firstcalibration image (including a grating pattern), and causes the secondprojector 150 b to output a second calibration image (including analignment pattern). In step S102, the image processing device 110 causesthe second projector 150 b to output a first calibration image(including both a grating pattern and an alignment pattern), and causesthe third projector 150 c to output a second calibration image(including an alignment pattern). In step S103, the image processingdevice 110 causes the third projector 150 c to output a firstcalibration image (including both a grating pattern and an alignmentpattern). In each of steps S101 through S103, the user takes images suchthat the calibration images being projected fit in the angular field ofthe camera 106, according to the guidance given by the image processingdevice 110, for example. In step S104, the image processing device 110collectively receives the input of a plurality of calibration-use imagesfrom the camera 160, and the process proceeds to step S109.

Meanwhile, in the second mode illustrated in FIG. 5, in step S105, theimage processing device 110 causes the first and third projectors 150 a,150 c to output a first calibration image (including a grating pattern),and causes the second projector 150 b to output a second calibrationimage (including an alignment pattern). In step S106, the imageprocessing device 110 receives input of the calibration-use image takenin step S105 from the camera 160. In step S107, the image processingdevice 110 causes the second projector 150 b to output a firstcalibration image (including both a grating pattern and an alignmentpattern). In step S108, the image processing device 110 receives theinput of the calibration-use image taken in step S107 from the camera160, and the process proceeds to step S109.

In step S109, for which details are described below, the imageprocessing device 110 calculates the geometric correction coefficientsof the respective projectors. In the geometric correction coefficientcalculation process for each projector in step S109, the imageprocessing device 110 extracts the feature points from the respectivecalibration-use images, converts the feature points into a commoncoordinate system of the grating points of the respectivecalibration-use images, and calculates the geometric correctioncoefficient of each projector. In step S110, for which details aredescribed below, the image processing device 110 calculates the blendingcoefficients of the respective projectors.

In step S111, the image processing device 110 sets, in the respectivecorrection processing units 114, the geometric correction coefficientsand the blending coefficients for each of the projectors, calculated insteps S109 and S110. In step S112, the image processing device 110causes the switching unit 122 to switch the input of the projected imageoutput unit 116 to the output of the correction processing unit 114, andshifts to the projection mode.

In step S113, the image processing device 110 reads the content image.In step S114, the image processing device 110 executes a correctionprocess on the content image by the correction processing unit 114 ofeach projector. In step S115, the image processing device 110 causes theprojected image output unit 116 of each projector to output thecorrected projection image of each projector.

In step S116, the image processing device 110 determines whether aninstruction to end the projection mode has been received from the user.In step S116, when the image processing device 110 determines that aninstruction to end the projection mode has not been received (NO), theprocess loops to step S113, and the projection image is updated. In thecase of a video, the process proceeds to a process for the next frame.In step S116, when the image processing device 110 determines that aninstruction to end the projection mode has been received (YES), theprocess is branched to step S117, and the process ends.

Calculation of Geometric Correction Coefficient

In the following, with reference to FIGS. 7 through 13 and 17A, adescription is given of details of the process of calculating geometriccorrection coefficients of the respective projectors. FIG. 7 is aflowchart indicating a process of calculating a geometric correctioncoefficient executed by the correction coefficient calculation unit 130according to the present embodiment. The process illustrated in FIG. 7is started from step S200 when the process is called in step S109 ofFIG. 6.

In step S201, the feature point extraction unit 132 extracts, from eachof the plurality of calibration-use images that have been prepared, thegravity center coordinates of the circles in the projected image of eachprojector 150 in each of the taken image coordinate systems, as gratingpoint coordinates (decimal point accuracy). The gravity centercoordinates of the circles may be calculated by, for example, binarizingthe image, cutting out a bundle of white pixels by pattern matching, andobtaining the gravity center coordinates of the bundle of white pixels.

In step S202, the feature point extraction unit 132 extracts, from eachof the plurality of calibration-use images, the gravity centercoordinates of the rectangular markers of the projected images of theprojectors 150 in each of the taken image coordinate systems, asalignment point coordinates. Similarly, the gravity center coordinatesof the rectangular markers may be calculated by, for example, binarizingthe image, cutting out a bundle of white pixels by pattern matching, andobtaining the gravity center coordinates of the bundle of white pixels.

A detailed description is given of the first mode illustrated in FIG. 4.The feature point extraction unit 132 extracts, from the respectivecalibration-use images obtained by taking the n (1≦n≦N−1)th calibrationprojection scene, in the arrangement of projected images, the alignmentpoint of the alignment pattern (not necessarily projected by firstprojector) and at least the grating points of the grating patternprojected by one (nth projector) of projectors arranged adjacent to eachother. Furthermore, the alignment points of the alignment patternprojected by the other one of the projectors (n+1th projector). From thecalibration-use image obtained by taking the last Nth calibrationprojection scene, the grating points of the grating pattern and thealignment points of the alignment pattern projected by the lastprojector (Nth) are extracted.

Note that in a single calibration-use image, the circular patterns ofone of the projectors and the four alignment rectangular markers of theother projector may be identified by using the positional relationshipbetween each other. When the calibration projection scene is constitutedaccording to the first mode illustrated in FIG. 4, there are rectangularmarkers outside the circular patterns, and eight rectangular markers ofprojectors arranged adjacent to each other on the left and right arearranged in the order of two left rectangular markers of the leftprojector, two left rectangular markers of the right projector, tworight rectangular markers of the left projector, and two rightrectangular markers of the right projector. Based on such a positionalrelationship, it is possible to identify each of the circular patternsand rectangular markers. Note that other than using a positionalrelationship, for example, the color and shape of the rectangularmarkers in the taken image may be identified by changing the color andshape of the rectangular markers to be projected for each projector, andthe determination may be made based on the identified features.

A description is given of a second mode illustrated in FIG. 5. Thefeature point extraction unit 132 extracts, from the calibration-useimage obtained by taking the first calibration projection scene, in thearrangement of projected images, the grating points of the gratingpattern projected by projectors of the first group (for example, theprojectors in the odd number order). The feature point extraction unit132 extracts, from the calibration-use image obtained by taking thesecond calibration projection scene, the grating points of the gratingpattern projected by projectors of the second group (for example, theprojectors in the even number order). Furthermore, the feature pointextraction unit 132 extracts, from the respective calibration-use imagesobtained by taking the first calibration projection scene and the secondcalibration projection scene, the alignment points of the alignmentpattern projected by a common projector 150.

In step S203, the grating point conversion unit 134 calculates aprojection conversion coefficient for a predetermined pair ofcalibration-use images, based on the alignment point coordinates of therectangular markers common to the taken images. In step S204, thegrating point conversion unit 134 converts the grating point coordinatesof the projected images of the respective projectors into a commoncoordinate system, and combines the grating point coordinates, based onthe projection conversion coefficient calculated in step S203.

FIG. 8 is for describing three calibration-use images prepared by takingimages of the calibration projection scenes, and the projectionconversion coefficient that is calculated among these taken images, inthe first mode. FIG. 9 is for describing two calibration-use imagesprepared by taking images of the calibration projection scenes, and theprojection conversion coefficient that is calculated among these takenimages, in the second mode.

In the first mode, as illustrated in FIG. 8, with respect to the pair ofthe first and second calibration-use images 250, 260, the pair of thealignment point coordinates are obtained, which are of the rectangularmarkers 254, 264 of the projected images 252 b, 262 b of the secondprojector 150 b common to the taken images. Then, based on this pair ofalignment point coordinates (254, 264), the grating point conversionunit 134 calculates the projection conversion coefficient for convertingthe coordinate system of the second calibration-use image 260 into thecoordinate system of the first calibration-use image 250. Similarly,with respect to the pair of the second and third calibration-use images260, 270, the pair of the alignment point coordinates are obtained, ofthe rectangular markers 264, 274 of the projected images 262 c, 272 c ofthe third projector 150 c common to the taken images. Based on this pairof alignment point coordinates (264, 274), the grating point conversionunit 134 calculates the projection conversion coefficient for convertingthe coordinate system of the third calibration-use image 270 into thecoordinate system of the second calibration-use image 260.

The conversion formula of projection conversion is expressed by thefollowing Formula (1), and by eliminating the denominator and organizingFormula (1), Formula (1) can be expanded into a first-degree polynomialequation of Formula (2).

Formula  (1) $\begin{matrix}{{u = \frac{{x*a} + {y*b} + c}{{x*g} + {y*h} + 1}}{v = \frac{{x*d} + {y*e} + f}{{x*g} + {y*h} + 1}}{{Formula}\mspace{14mu}(2)}} & (1) \\{{u = {{x*a} + {y*b} + c - {x*g*u} - {y*h*u}}}v = {{x*d} + {y*e} + f - {x*g*v} - {y*h*v}}} & (2)\end{matrix}$

In the above Formulas (1) and (2), x, y express the planar coordinatesbefore conversion, u, v express the planar coordinates after conversion,and the eight coefficients of a through h express projection conversioncoefficients. In the above formulas, in order to calculate eightprojection conversion coefficients which are unknown parameters, atleast eight simultaneous equations are required; however, if there arefour corresponding points of alignment in the two calibration-use imagesdescribed above, eight conversion formulas can be generated. By solvingthe eight simultaneous equations, generated from the correspondingpoints of the four rectangular markers, it is possible to obtain theprojection conversion coefficients a through h.

In the first mode, when the projection conversion coefficients a throughh between the two pairs of taken images are calculated, the gratingpoint conversion unit 134 executes projection conversion of convertingthe extracted grating points of the second calibration-use image intothe coordinate system of the first taken image. Furthermore, the gratingpoint conversion unit 134 executes projection conversion of convertingthe extracted grating points of the third calibration-use image, fromthe coordinate system of the third taken image into the coordinatesystem of the second taken image, and further executes projectionconversion of converting the coordinate system of the second taken imageinto the coordinate system of the first taken image. Accordingly, thegrating point coordinates of all of the projectors 150 a through 150 care converted into a common coordinate system that is the coordinatesystem of the first calibration-use image taken by directly facing thescreen, and are combined together.

In the second mode, as illustrated in FIG. 9, with respect to the pairof the first and second calibration-use images 280, 290, the pair of thealignment point coordinates are obtained, which are of the rectangularmarkers 284, 294 of the projected images 282 b, 292 b of the secondprojector 150 b common to the taken images. Then, based on this pair ofalignment point coordinates, the projection conversion coefficient iscalculated, which is for converting the coordinate system of the secondtaken image 290 into the coordinate system of the first taken image 280.Based on the projection conversion coefficients a through h between oneset of the calibration-use images, the grating point conversion unit 134executes projection conversion of converting the extracted gratingpoints of the second calibration-use image into the coordinate system ofthe first taken image, and converts the grating points into a commoncoordinate system.

FIG. 10 schematically illustrates an assembly of grating pointcoordinates of the projectors combined on a common coordinate system300. As illustrated in FIG. 10, the assemblies of grating pointcoordinates of the projectors 302 a, 302 b, and 302 c are converted ontothe common coordinate system 300 of the first calibration-use image, andare combined. Note that in FIG. 10, the circles of the plurality ofprojectors 150 a through 150 c for which images have been taken areexpressed as overlapping each other; however, there is no need for theimages per se to be overlapped.

Referring back to FIG. 7, in step S205, for each of the projectors 150,the geometric correction coefficient calculation unit 136 performslinear extrapolation on the grating point coordinates, which have beenconverted to the common coordinate system and combined, and calculatesthe outer periphery coordinates of the area where projection is possible(projection possible area).

FIGS. 11A and 11B illustrate a method of calculating the outer peripherycoordinates of the projection possible area (an area where projection ispossible) according to linear extrapolation by using the grating pointcoordinates that have been combined. FIG. 11A illustrates the fourgrating points in the top left corner in the projector memory, and FIG.11B illustrates the corresponding four grating points on the commoncoordinate system. As illustrated in FIG. 11A, the outer peripherycoordinates in the projector memory (grating points in four corners andalong four sides) are defined at a position where the quadrilateralpatch of four grating points (for example, P00_(P) through P11_(P))positioned on the outer periphery, is extrapolated (a position at adistance that is 1.5 times that of the distance between grating points).

The coordinates of the outer periphery pixels (grating points in fourcorners and along four sides) corresponding to the projection possiblearea of each of the projectors in the common coordinate system, can becalculated by linearly extrapolating points from the four grating pointcoordinates positioned on the outer peripheral part, as illustrated inFIG. 11B. Similarly, the points on the common coordinate systemcorresponding to arbitrary coordinate points in the projector memoryother than the outer periphery coordinates (grating points in fourcorners and along four sides) can be obtained by linearly extrapolatingor interpolating points from the four grating point coordinates that arenearby.

It is assumed that an arbitrary coordinate point Q_(P) in the projectormemory is a point of internal division in the x axis direction by t:1−t(0<t<1) and in the y axis direction by s:1−s (0<s<1), in the fourgrating points P00_(P), P10_(P), P01_(P), P11_(P) whose coordinatepositions are nearby in the projector memory. Then, a point Q_(C) in thecommon coordinate system corresponding to the coordinate point Q_(P),can be calculated by using the following Formula (3), from thecoordinate vectors of the corresponding four grating points P00_(C),P10_(C), P01_(C), P11_(C). In the case of a point that is to beextrapolated, the point Q_(C) can be calculated by setting the ranges of−1.5<t<0, −1.5<s<0 with respect to the above t and s, and using thefollowing Formula (3).Formula (3)Q _(C)=(1−s)((1−t)/P00_(C) +tP10_(C))+s((1−t)P01_(C) +tP11_(C))  (3)

In the entire image, a non-linear geometric distortion may occur;however, in this case, it is assumed that the distortion is a lineargeometric distortion in parts of the image, including the range of thequadrilateral patch constituted by grating points of 2×2, and the rangewhere a predetermined amount of points have been extrapolated toward theouter periphery. This is because the size of the above quadrilateralpatch can be deemed as being sufficiently small. Note that in thedescribed embodiment, it is assumed that the corresponding points arecalculated by linear interpolation by using the above Formula (3).However, in other embodiments, the point Q_(P) in the projector memorycan be associated with the corresponding point Q_(C) in the commoncoordinate system, by projection conversion obtained by using fouradjacent pairs of grating points P00_(C), P10_(C), P01_(C), P11_(C),P00_(P), P10_(P), P01_(P), P11_(P).

By performing the linear extrapolation described above for eachprojector, the projection possible areas of the three projectors 150 athrough 150 c (i.e., the range where a white image can be entirelyprojected) are detected in the common coordinate system. FIG. 12 (A)expresses the projection possible areas 304 a through 304 c of threeprojectors detected in the common coordinate system 300. The projectionpossible area 304 a of the first projector 150 a is indicated by a solidwhite line, the projection possible area 304 b of the second projector150 b is indicated by a dashed white line, and the projection possiblearea 304 c of the third projector 150C is indicated by a dashed-twodotted line.

Referring back to FIG. 7, in step S206, the geometric correctioncoefficient calculation unit 136 obtains the logical sum (OR) of theprojection possible areas of all projectors in the common coordinatesystem, and sets, in the area of the above logical sum, a projectiontarget area after correction for mapping the content image. Theprojection target area after correction is set such that the contentimage can be mapped by the maximum size in the area that is the logicalsum of the projection possible areas 304 a through 304 c of allprojectors, while maintaining the aspect ratio.

The points of the four corners in each of the projection possible areasin the common coordinate system are known, and the four sides connectingthese points (top side, bottom side, left side, right side) are obtainedin a form of being linearly divided by the grating point width, and therange including these sides is recognized. Therefore, the rectangularrange, which may be formed within the area of the three logical sums, isdefined in a range sandwiched between the top side 306T and the bottomside 306B, and in the range sandwiched between the left side 306L andthe right side 306R, of the projection possible areas 304 a through 304c of the three projectors in the common coordinate system.

As indicated by the rectangular area indicated by a dashed line in FIG.12 (A), the projection target area after correction 310 is an area thatis assigned by the maximum size in the rectangular range having the foursides 306T, 306B, 306L, and 306R, while maintaining the aspect ratio(for example, M:N) of the content image. In the example of FIG. 12 (A),there are slight blank spaces in the vertical direction, and thereforemargins are provided at the top and bottom, and the projection targetarea after correction is centered. Then, as illustrated in FIG. 12 (B),in the projection target area after correction 310, the content image320 to be projected is pasted.

Referring back to FIG. 7, in the loop of steps S207 through S211, theprocesses of steps S208 through S210 are executed for each projector,and the geometric correction coefficients for each of the plurality ofprojectors are obtained. In step S208, the geometric correctioncoefficient calculation unit 136 converts the grating point coordinatesin the common coordinate system to the coordinate system of the originalcontent image. In the following, the content image to be pasted to theprojection target area after correction 310 in the common coordinatesystem is referred to as a “projection content image”, and the originalcontent image that is the source of the projection content image isreferred to as an “equal-magnification content image”.

In step S209, the geometric correction coefficient calculation unit 136associates the grating point coordinates in the projector memory withthe pixel positions in the coordinate system of the equal-magnificationcontent image, via the common coordinate system. In step S210, thegeometric correction coefficient calculation unit 136 associates theinteger pixel coordinates in the projector memory with the pixelpositions in the coordinate system of the equal-magnification contentimage by linear interpolation, via the common coordinate system.

As illustrated in FIG. 13, the geometric correction coefficientcalculated by the process of steps S208 through S210 are for associatingthe coordinates in a projector memory 330 with pixel positions in theequal-magnification content image corresponding to the positions in theprojection content image.

A description is given of one grating point P42_(P) in the projectormemory 330 a illustrated in FIG. 13, as a representative example. Withrespect to a grating point P42_(P) in the projector memory 330, acorresponding point P42_(C) (X_(P42C), Y_(P42C)) in the commoncoordinate system 300 is extracted. Then the content image is mapped inthe projection target area after correction 310, and therefore asillustrated in FIG. 13, with respect to the coordinate position P42_(C)on the common coordinate system 300, a corresponding pixel positionP42_(m) (X_(P42m) Y_(P42m)) in the equal-magnification content image isfurther defined.

The corresponding pixel position P42_(m) (X_(P42m), Y_(P42m)) on theequal-magnification content image can be calculated by the followingFormula (4) from the coordinates (X_(P42C) Y_(P42C)) of thecorresponding point P42_(C) on the common coordinate system 300. In thefollowing Formula (4), the coordinates (X₀, Y₀) are coordinates of theorigin point at the top left of the projection content image on thecommon coordinate system, and R expresses the magnification ratio of thecontent image. Note that in this example, as a matter of convenience,the equal-magnification content image is assumed to be directly mappedon the projection target area after correction 310 by a predeterminedmagnification ratio R; however, the method of mapping the content on thecommon coordinate system is not particularly limited.Formula (4)X _(P42m)=(X _(P42C) −X ₀)/RX _(P42m)=(Y _(P42C) −Y ₀)/R  (4)

Similarly, with respect to all of the grating points Pij_(P) other thanthe grating point P42_(P) in the projector memory, the correspondingpixel position on the equal-magnification content image is calculated.As for arbitrary coordinates other than the grating points in theprojector memory, the corresponding pixel position in theequal-magnification content image can be calculated by the same methodas that described with reference to FIG. 11, by linearly interpolating(interpolating or extrapolating at the peripheral part) thecorresponding pixel position on the content image of a nearby 2×2grating point. Accordingly, the pixel position of the area 322 a handledby the first projector 150 a in the content image 320 is associated withthe pixel of a predetermined area 332 a in the projector memory 330 a.

FIG. 17A illustrates an example of a data structure of a geometriccorrection coefficient of one projector calculated by the process ofsteps S208 through S210. As illustrated in FIG. 17A, the correspondingpixel position on the equal-magnification content image with respect toall pixels in the projector memory obtained as above, becomes thegeometric correction coefficient.

The loop of steps S207 through S211 is repeated for the number ofprojectors, and when the association of the integer pixel coordinates inthe projector memory and the coordinate system of theequal-magnification content image is completed for all of theprojectors, the process proceeds to step S212. In step S212, the processis ended, and the process returns to the call source indicated in FIG.8. Accordingly, a geometric correction coefficient is prepared for allof the respective projectors 150 a through 150 c.

Note that in the described embodiment, the corresponding pixel positionon the equal-magnification content image is obtained for all pixels inthe projector memory, as geometric correction coefficients; however, thepresent embodiment is not so limited. In other embodiments, pixelpositions Pij_(m) on the equal-magnification content image, with respectto the grating points Pij_(P) in the projector memory, are obtained asthe geometric correction coefficients, and the correction processingunit 114 described below may calculate the coordinates other than thegrating points by performing projection conversion or linear conversionfor each quadrilateral patch.

Calculation of Blending Coefficient

With reference to FIGS. 14 through 17, a description is given of detailsof the process of calculating the blending coefficient of eachprojector. FIG. 14 is a flowchart of a process of calculating a blendingcoefficient executed by the correction coefficient calculation unit 130,according to the present embodiment. The process of FIG. 14 is startedfrom step S300 when the process is called in step S110 of FIG. 6. In theloop of steps S301 through S313, the processes of steps S302 throughS312 are executed for each target projector, and a blending coefficientis obtained for each of the plurality of projectors 150 a through 150 c.

In step S302, the blending coefficient calculation unit 138 detects theoverlapping area of the target projector and a projector adjacent to thetarget projector in the common coordinate system 300, based on the outerperipheral coordinates of the projection possible areas of theseprojectors. FIG. 15 is for describing the association of blendingcoefficients with respect to the coordinates in the projector memory330. As illustrated in FIG. 15, in the top side of the projection targetarea after correction 310 in the common coordinate system 300, bysearching from the left origin point (◯) to the right direction, thestarting point (●) and the ending point (⊚) of the overlapping area ofthe first projector 150 a and the second projector 150 b are detected.Similarly, for other horizontal lines, the starting point and the endingpoint of the overlapping area are detected.

Referring back to FIG. 14, in step S302, the blending coefficientcalculation unit 138 first initializes the blending coefficients withrespect to the coordinates of the common coordinate system, by zero. Inthe loop of steps S304 through S311, the processes of steps S305 throughS310 are executed for each of the horizontal lines in the commoncoordinate system (only the part corresponding to the projection targetarea after correction). By the processes of steps S305 through S310, theintermediate result of the blending coefficient is assigned to eachcoordinate position on the common coordinate system.

In step S305, in the target horizontal line, based on the aboveperipheral coordinates of the projection possible area and the detectedoverlapping area, the starting point and the ending point of theprojection possible area of the projector, and the starting point andthe ending point of the overlapping area of the projector and theadjacent projector, are set.

In the loop of steps S306 through S310, the processes of steps S307through S309 are executed for each pixel in the horizontal line of thecommon coordinate system (only inside the projection possible area). Bythe processes of steps S307 through S309, a blending coefficient isdetermined for each pixel on the common coordinate system in thehorizontal line.

In step S307, the blending coefficient calculation unit 138 branches theprocess according to whether the target pixel corresponds to theoverlapping area. In step S307, when the blending coefficientcalculation unit 138 determines that the target pixel does notcorrespond to the overlapping area (NO), the process proceeds to stepS308. In this case, the pixel corresponds to a single projectionpossible area that does not overlap with other areas, and therefore instep S308, the blending coefficient calculation unit 138 determines theblending coefficient to be the maximum value 1. Meanwhile, in step S307,when the blending coefficient calculation unit 138 determines that thetarget pixel corresponds to the overlapping area (YES), the processproceeds to step S309. In this case, the pixel corresponds to an areaoverlapping with the adjacent projector, and therefore in step S309, theblending coefficient calculation unit 138 calculates the blendingcoefficient according to a predetermined relational expression.

FIG. 16 illustrates a graph of input output properties of a projector;however, the input output properties of a projector are typically notlinear. In the calculation of the blending coefficient for the pixelcorresponding to the above overlapping area, inverse correction is firstperformed on the input output properties so that the input outputproperties become linear, and then weighting is performed such that thelight amount from the projectors on both sides becomes a total of one.

Specifically, as indicated for the first projector in the graph at thebottom of FIG. 15, for the pixels in the range from the origin point (◯)to the starting point (●) of the overlapping area, the blendingcoefficient is determined to be a maximum of one in step S308 describedabove. Meanwhile, for the pixels in the range from the starting point(●) to the ending point (⊚) of the overlapping area, in step S309, theblending coefficient is calculated by performing inverse correction onthe input output properties of the projector, such that the actualbrightness gradually decreases from 1.0 to zero in a linear manner,according to the horizontal distance from the starting point (●). If theinput output properties are as illustrated in FIG. 16, the blendingcoefficient y for the horizontal distance x (0.0≦x≦1.0) from thestarting point, which is normalized by the distance from the startingpoint to the ending point, can be calculated by the following Formula(5).Formula (5)y=1.0−x ^(0.5)  (5)

By the loop of steps S304 through S311, the intermediate result of theblending coefficient is determined for each of the integer pixels in thecommon coordinate system. In areas other than the projection possiblearea, zero is set by the initialization process of step S303. Whenprocesses for all horizontal lines in the common coordinate system arecompleted by the loop of steps S304 through S311, the process proceedsto step S312. With respect to the horizontal lines outside theprojection target area after correction, the pixels are set to zero bythe initialization process of step S303.

In step S312, the blending coefficient calculation unit 138 associates,to the respective integer pixel coordinates in the projector memory, theblending coefficient assigned to the nearest integer pixel among thecoordinates (decimal points) of the common coordinate system associatedby the data structure illustrated in FIG. 17A. FIG. 17B illustrates anexample of a data structure of blending coefficients of a singleprojector calculated by the process of steps S302 through S312. Asillustrated in FIG. 17B, the blending coefficients of all pixels of theprojector memory are obtained.

When the process for all projectors are completed by the loop of stepsS301 through S313, the present process is ended in step S314, and theprocess returns to the call source indicated in FIG. 6.

By the above process, for each of the plurality of projectors 150 athrough 150 c, blending coefficients for all pixels of the projectormemory are obtained. Note that in the above description, the overlappingarea of the first projector 150 a and the second projector 150 b isdescribed. When the second projector 150 b is the target, the firstprojector 150 a and the third projector 150 c on the left and right arecombined, and blending coefficients for the two overlapping areas arecalculated.

Correction Process

In the following, with reference to FIGS. 17A through 18, a descriptionis given of details of the correction process based on the abovecorrection coefficient. FIG. 18 is for describing a correction processbased on the above correction coefficient. The above-described geometriccorrection coefficients of the projectors calculated by the geometriccorrection coefficient calculation unit 136, and the above-describedblending coefficients of the projectors calculated by the blendingcoefficient calculation unit 138, are set in the respective correctionprocessing units 114 in step S111 of FIG. 6.

First, the correction processing unit 114 prepares the association datafor associating all of the pixels of the projector memory with thecorresponding pixel positions on the equal-magnification content image.When the pixel positions with respect to all pixels of the projectormemory as illustrated in FIG. 17A have been obtained by the process bythe geometric correction coefficient calculation unit 136 describedabove, the correction processing unit 114 directly reads the associationdata illustrated in FIG. 17A. When only the pixel positions on theequal-magnification content image for each grating point coordinate ofthe projector memory are given, the coordinates on theequal-magnification content image to be referred are calculated bylinear interpolation from the coordinates of the grating points for allpixels in the projector memory other than the grating points, and theassociation data as illustrated in FIG. 17A is calculated.

The correction processing unit 114 generates an intermediate image fromthe equal-magnification content image to be projected, by a pixelinterpolation method such as bi-linear and bi-cubic, based on the pixelpositions (decimal points) on the equal-magnification content image tobe referred to for each pixel in the projector memory. Furthermore, thecorrection processing unit 114 multiplies the pixel values of therespective colors R, G, B in the generated intermediate image, by theblending coefficient associated by the association data of FIG. 17B, andgenerates the final projection image.

FIG. 18 illustrates projection images 350 a through 350 c which havebeen finally obtained from the content image for the three projectors150 a through 150 c, by the correction processing units 114 a through114 c. As illustrated in FIG. 18, in the projection mode, theseprojection images 350 a through 350 c are projected from the projectors150. In the projection image 350, the part of the content image to behandled by the corresponding projector 150 is subjected to variouscalibrations, and therefore the projected images of the projectionimages 350 a through 350 c are superposed on the projection surface in apreferable manner, and combined into a single projected image 352.

Modification Example of Calibration Scene Selection Unit

In the following, with reference to FIGS. 19A through 23, a descriptionis given of a modification example of the embodiment. In the above, adescription is given of arranging projected images in a row in ahorizontal direction or a vertical direction and performingmulti-projection. In FIGS. 19A through 23, a description is given bygeneralizing to multi-projection in which projected images are arrangedin a two-dimensional grating form (ij).

In this modification example of the embodiment also, the calibrationscene selection unit 120 reads the respective calibration images fromthe calibration image storage unit 118, and selects an appropriatecalibration image and outputs the selected calibration image to theplurality of projectors 150 a through 150 c. In this modificationexample of the embodiment, there are two types of calibration images asillustrated in FIG. 19A; a first calibration image Cij including only agrating pattern, and a second calibration image Aij including only analignment pattern. The alignment pattern is preferably arranged at aposition around the area where the grating pattern is arranged. A thirdcalibration image is also used, which is provided as a calibration imageCij+Aij obtained by combining the first calibration image Cij and thesecond calibration image Aij.

The calibration scene selection unit 120 has recognized the positionalrelationships of the projected images of the plurality of projectors ij,and in order to obtain calibration results of the projectors 150 overallwithout deficiencies, and the calibration scene selection unit 120prepares the plurality of calibration projection scenes such that thefollowing conditions (A) through (D) are satisfied.

The first condition (A) is a condition that in the arrangement ofprojected images, projectors 150 that are adjacent to each other do notproject grating patterns at the same time in the same scene. That is tosay, as illustrated in FIG. 19B, when the grating pattern Cij isprojected from a projector ij in one scene, the grating patterns of theeight projectors adjacent to the projector ij cannot be projected. Bypreparing a plurality of calibration projection scenes to satisfy thefirst condition (A), it is possible to prevent the grating patterns ofadjacent projectors from overlapping each other.

The second condition (B) is a condition that in all of the plurality ofcalibration projection scenes, at least one grating pattern Cij of allprojectors participating the multi-projection, is included. By preparinga plurality of calibration projection scenes to satisfy the secondcondition (B), it is possible to ensure that distortion correction isperformed on the projected image of all projectors ij.

The third condition (C) is a condition that one calibration projectionscene includes an alignment pattern Aij projected from a projector ijthat is common to the one calibration projection scene and at least oneof the other calibration projection scenes. The fourth condition (D) isa condition based on the assumption of the third condition (C), and thatin all of the plurality of calibration projection scenes, when scenesare connected based on an alignment pattern Aij common to the scenes,the calibration projection scenes are used as nodes and theabove-described connection is used as a link to form a single treestructure. Forming a tree structure by using the calibration projectionscenes as nodes and using the connection as a link, means that thescenes can be combined in a coordinate system of a calibration-use imageobtained by taking an image of one scene that is the route. Therefore,by preparing a plurality of calibration projection scenes to satisfy thethird condition (C) and the fourth condition (D), it is ensured that thecoordinates of the calibration-use images obtained by taking images ofall of the scenes can be combined in the common coordinate system.

FIGS. 20 and 21 are for describing the calibration projection scenesformed to satisfy the conditions (A) through (D), and a method of takingimages of the scenes. FIG. 20 illustrates an example where threeprojected images are connected in the horizontal direction, and FIG. 21illustrates an example where three projected images are connected in thevertical direction.

In the mode illustrated in FIG. 20, three calibration projection scenesare prepared, which correspond to three taken images. The threecalibration projection scenes 402, 412, 422 are linked by alignmentpatterns A10 and A20 which are included in and common to the calibrationimages 400 b, 410 b and calibration images 410 c, 420 c, and constitutea single tree structure T1. In the mode of FIG. 20, it is necessary totake images of the scenes three times; however, the camera 160 canperform imaging by an angular field in which projected images of twoprojectors can fit.

In the mode of FIG. 21, two calibration projection scenes are prepared,corresponding to images taken two times. The two calibration projectionscenes 432, 442 are linked by an alignment pattern A01 which is includedin and common to the calibration images 430 b, 440 b, and constitute asingle tree structure T2. In the mode of FIG. 21, the camera 160 needsto perform imaging by an angular field in which projected images ofthree projectors can fit; however, images of the scenes only need to betaken two times.

FIGS. 22 and 23 are for describing the calibration projection scenesformed to satisfy the conditions (A) through (D), and a method of takingimages of the scenes. FIGS. 22 and 23 respectively illustrate an examplewhere nine projectors are used to connect the projected images in threelines and three rows. FIG. 22 illustrates an example in which the entirescreen of 3×3 is fit in the angular field of the camera 160 and an imageof the scene is taken four times. FIG. 23 illustrates an example inwhich two screens of 1×2 or 2×1 are is fit in the angular field and animage of the scene is taken nine times.

In the mode of FIG. 22, four calibration projection scenes are prepared,which correspond to four taken images. In the calibration projectionscenes 450, 452, 454, 456 illustrated in FIG. 22, grating patterns arenot projected simultaneously from projectors that are adjacent to eachother, and the first condition (A) is satisfied. Furthermore, all of thecalibration projection scenes 450, 452, 454, 456 include one of each ofall of the grating patterns Cij (ij=00 through 22) of the projectors ij,and the second condition (B) is satisfied. Furthermore, by the alignmentpatterns common to the scenes indicated by A01, A10, and A11, the fourcalibration projection scenes 450, 452, 454, 456 constitute the treestructure T3, and the third condition (C) and the fourth condition (D)are satisfied. In the mode illustrated in FIG. 22, the camera 160 needsto perform imaging by an angular field in which projected images of 3×3projectors can fit; however, images of the scenes only need to be takenfour times.

Meanwhile, in the mode of FIG. 23, nine calibration projection scenesare prepared, which correspond to nine taken images. In the calibrationprojection scenes 460 through 468 illustrated in FIG. 23, in anarrangement of three lines and three rows, images are taken by fittingtwo projectors in the angular field in a unicursal manner from thecenter toward the outer periphery. Then, the n (1≦n≦N−1)th calibrationprojection scene is prepared, such that one of the projectors arrangedadjacent to each other projects an alignment pattern (the firstprojector does not necessarily need to project an alignment pattern) andat least a grating pattern, and the other one of the projectors projectsan alignment pattern Aij. Therefore, the calibration projection scenes460 through 468 have a tree structure, and the first to fourthconditions are satisfied. In the mode of FIG. 23, it is necessary totake images of the scenes nine times; however, the camera 160 canperform imaging by an angular field in which projected images of twoprojectors can fit.

Hardware Configuration

In the following, a description is given of a hardware configuration ofthe image processing device 110 according to the above embodiment, withreference to FIG. 24. The image processing device 110 is typicallyconstituted as a general-purpose computer. FIG. 24 illustrates ahardware configuration of a general-purpose computer according to thepresent embodiment.

The general-purpose computer 110 is, for example, a desktop personalcomputer or a workstation. The general-purpose computer 110 of FIG. 24includes a CPU (Central Processing Unit) 12, a north bridge 14 thathandles the connection of the CPU 12 and a memory, and a south bridge16. The south bridge 16 is connected with the north bridge 14 via anexclusive-use bus or a PCI bus, and handles the connection with I/O suchas the PCI bus and a USB memory.

To the north bridge 14, a RAM (Random Access Memory) 18 for providing awork area of the CPU 12, and a graphic board 20 for outputting imagesignals are connected. The graphic board 20 is connected to a display 50or the above projector 150, via an image output interface such as ananalog RGB, HDMI (High-Definition Multimedia Interface; HDMI andHigh-Definition Multimedia Interface are registered trademark ortrademark), DVI (Digital Visual Interface), DisplayPort (registeredtrademark)

To the south bridge 16, a PCI (Peripheral Component Interconnect) 22, aLAN port 24, IEEE 1394, a USB (Universal Serial Bus) port 28, asecondary storage device 30, an audio input output 32, and a serial port34 are connected. The secondary storage device 30 is, for example, a HDD(Hard Disk Drive) or a SSD (Solid State Drive), and stores an OS forcontrolling the computer device, programs for realizing the abovefunctional units, various kinds of system information, and various kindsof setting information. The LAN port 24 is an interface device forconnecting the general-purpose computer 110 to a network by wired orwireless connection.

To the USB port 28, an input device such as keyboard 52 and a mouse 54may be connected; the USB port 28 may provide a user interface forreceiving input of various instructions from the operator. Thegeneral-purpose computer 110 according to the present embodiment readsprograms from the secondary storage device 30 and loads the programs inthe work space provided by the RAM 18, to realize the functional unitsand the processes described above under the control of the CPU 12. Notethat the projector 150 and the camera 160 are not particularlydescribed, but also includes hardware such as a CPU and a RAM, andhardware according to particular purposes.

By the configuration of the embodiments described above, it is possibleto define a coordinate system of a projector memory (output image) amongprojectors that are adjacent to each other, to make it easy to avoid theoverlapping of grating patterns used for detecting a distortion in theprojected image. Thus, compared to a case that requires image processingfor pattern separation, patterns can be precisely extracted, andgeometric correction and blending correction can be performed with highprecision.

Furthermore, by providing the markers of the alignment pattern on theoutside of the grating pattern, it is easy to project the alignmentpattern and the grating pattern without overlapping each other. Thus, itis possible to highly-precisely combine the grating point coordinates ofthe calibration-use images, which have been taken in a divided manner(by dividing the image) over a plurality of times. Furthermore, thealignment pattern is used to combine the calibration-use images, whichhave been taken in a divided manner, and therefore there is no need tofix the camera with a tripod while taking the images in a dividedmanner. Furthermore, there is no need for any exclusive-use equipmentfor accurately controlling the position and orientation of the camera.Thus, the correction condition for a plurality of projectors can beeasily obtained at low cost under alleviated imaging conditions.

Furthermore, by devising the configuration of the scenes, gratingpatterns of a plurality of projectors can be taken in a divided manner,and therefore even if the number of screens in multi-projectionincreases, it is possible to avoid the restriction (of the position ofthe camera) in the depth direction when taking an image with a camera.Furthermore, when the requirements are not strict with respect to thedepth direction when taking images with the camera, by increasing thenumber of screens to be fit in the angular field of the camera, it ispossible to reduce the number of times of taking images for calibration,such that the man-hour for the calibration operation by the user can bereduced.

As described above, according to an embodiment of the present invention,a projection system, an image processing device, and a projection methodare provided, by which in the projection system for projecting images ona projection body by a plurality of projection units, the conditions forcorrecting images to be projected from the plurality of projection unitscan be obtained under alleviated imaging conditions.

Note that the above functional units can be realized by acomputer-executable program described in a legacy programming languagesuch as assembler, C, C++, C#, Java (registered trademark), or an objectoriented programming language, and may be distributed by being stored ina device-readable recording medium such as ROM, EEPROM, EPROM, a flashmemory, a flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, aBlu-ray disc, an SD card, an MO, or through an electric communicationline. Furthermore, part of or all of the above functional units may beimplemented on as programmable device (PD) such as a field programmablegate array (FPGA), or may be implemented as ASIC (application specificintegrated circuit), and may be distributed by a recording medium ascircuit configuration data (bit stream data) to be downloaded to the PDfor realizing the functions on the PD, and data described in HDL(Hardware Description Language), VHDL (VHSIC (Very High Speed IntegratedCircuits) Hardware Description Language)), and Verilog-HDL forgenerating the circuit configuration data.

According to one embodiment of the present invention, in a projectionsystem for projecting an image on a projection body by a plurality ofprojection units, the correction condition for an image to be projectedfrom the plurality of projection units, can be obtained under alleviatedimaging conditions.

The projection system, the image processing device, and the projectionmethod are not limited to the specific embodiments described herein, andvariations and modifications may be made without departing from thespirit and scope of the present invention.

The present application is based on and claims the benefit of priorityof Japanese Priority Patent Application No. 2013-155719, filed on Jul.26, 2013, the entire contents of which are hereby incorporated herein byreference.

What is claimed is:
 1. A projection system, comprising: a plurality ofprojection units configured to project an image on a projection body; ataken image preparation unit configured to prepare a plurality ofcalibration-use images; an extraction unit configured to extract, fromeach of the plurality of calibration-use images, at least grating pointsindicating a distortion in a projected image of one of the plurality ofprojection units and alignment points of the projected image of the oneof the plurality of projection units or a projected image of another oneof the plurality of projection units; a conversion unit configured toconvert, onto a common coordinate system, the grating points of theprojected images of the plurality of projection units extracted from theplurality of calibration-use images by the extraction unit, based onalignment points common to the plurality of calibration-use images; ageometric correction coefficient calculation unit configured tocalculate a geometric correction coefficient for providing a projectionimage to be projected from the plurality of projection units, based onthe grating points on the common coordinate system; an image output unitconfigured to output, to at least one of the plurality of projectionunits, a calibration image including both of or one of a grating patterndefining grating points of the projected image and an alignment patterndefining alignment points between the plurality of calibration-useimages; and a scene preparation unit configured to prepare a pluralityof calibration projection scenes each including the calibration image tobe output to at least one of the plurality of projection units, suchthat (a) in an arrangement of the projected images, projection unitsadjacent to each other among the plurality of projection units do notboth project a grating pattern in the same calibration projection scene,(b) all of the plurality of calibration projection scenes include atleast one grating pattern of the plurality of projection units, and (c)the plurality of calibration projection scenes constitute a treestructure by a connection based on an alignment pattern projected by oneof the plurality of projection units common to the plurality ofcalibration projection scenes.
 2. The projection system according toclaim 1, wherein the scene preparation unit prepares one or more of theplurality of calibration projection scenes, such that in an arrangementof the projected images, one of the projection units adjacent to eachother projects at least a grating pattern, and another one of theprojection units adjacent to each other projects an alignment pattern,and the scene preparation unit prepares a last one of the plurality ofcalibration projection scenes, such that a last one of the plurality ofprojection units projects both of a grating pattern and an alignmentpattern.
 3. The projection system according to claim 1, wherein thescene preparation unit prepares a first one of the plurality ofcalibration projection scenes, such that in an arrangement of theprojected images, projection units in a first group among the pluralityof projection units project at least a grating pattern, the scenepreparation unit prepares a second one of the plurality of calibrationprojection scenes, such that projection units in a second group amongthe plurality of projection units, which are each adjacent to one of theprojection units in the first group, project at least a grating pattern,and the scene preparation unit prepares the first one and the second oneof the plurality of calibration projection scenes, such that at least acommon one of the plurality of projection units, which is common to thefirst one and the second one of the plurality of calibration projectionscenes, projects an alignment pattern.
 4. The projection systemaccording to claim 1, wherein the alignment pattern is arranged at aperipheral position of an area where the grating pattern is arranged, inthe projected image.
 5. The projection system according to claim 1,wherein the conversion unit performs projection conversion based on acoordinate of an alignment pattern common to a first calibration-useimage and a second calibration-use image among the plurality ofcalibration-use images prepared by the taken image preparation unit, theprojection conversion being performed for converting the grating pointsfrom a coordinate system of the second calibration-use image to acoordinate system of the first calibration-use image.
 6. The projectionsystem according to claim 1, further comprising: a blending coefficientcalculation unit configured to detect, for each of the plurality ofprojection units, an overlapping area between a projected image of atarget projection unit and a projected image of an adjacent projectionunit adjacent to the target projection unit, and calculate a blendingcoefficient for adjusting an overlapping amount between the projectedimage of the target projection unit and the projected image of theadjacent projection unit.
 7. The projection system according to claim 6,further comprising: a correction processing unit provided for each ofthe plurality of projection units, the correction processing unit beingconfigured to generate an intermediate image for each of the pluralityof projection units from an image that is a target of projection, basedon the geometric correction coefficient calculated by the geometriccorrection coefficient calculation unit, and to calculate the projectionimage for each of the plurality of projection units from theintermediate image, based on the blending coefficient calculated by theblending coefficient calculation unit.
 8. The projection systemaccording to claim 1, further comprising: both of or one of a pluralityof projection devices acting as the plurality of projection units; andan imaging device configured to take images of the plurality ofcalibration-use images.
 9. An image processing device for performingprojection with the use of plurality of projection units, the imageprocessing device comprising: a taken image preparation unit configuredto prepare a plurality of calibration-use images; an extraction unitconfigured to extract, from each of the plurality of calibration-useimages, at least grating points indicating a distortion in a projectedimage of one of the plurality of projection units and alignment pointsof the projected image of the one of the plurality of projection unitsor a projected image of another one of the plurality of projectionunits; a conversion unit configured to convert, onto a common coordinatesystem, the grating points of the projected images of the plurality ofprojection units extracted from the plurality of calibration-use imagesby the extraction unit, based on alignment points common to theplurality of calibration-use images; and a geometric correctioncoefficient calculation unit configured to calculate a geometriccorrection coefficient for providing a projection image to be projectedfrom the plurality of projection units, based on the grating points onthe common coordinate system, wherein the extraction unit extracts, fromeach of the plurality of calibration-use images, both of or one of thegrating points and the alignment points, wherein each of the pluralityof calibration-use images includes at least one calibration-useprojected image including both of or one of a grating pattern defininggrating points of the projected image and an alignment pattern definingalignment points between the plurality of calibration-use images, andthe taken image preparation unit prepares the plurality ofcalibration-use images each including the at least one calibration-useprojected image projected from at least one of the plurality ofprojection units, such that (a) in each of the plurality ofcalibration-use images, in an arrangement of the projected images,grating patterns projected by projection units adjacent to each otheramong the plurality of projection units are not both included, (b) allof the plurality of calibration-use images include at least one gratingpattern of the plurality of projection units, and (c) the plurality ofcalibration-use images constitute a tree structure by a connection basedon an alignment pattern, which is projected by one of the plurality ofprojection units common to the plurality of calibration-use images andwhose image is taken.
 10. A projection method of projecting an image ona projection body by a plurality of projection units, the projectionmethod comprising: preparing, by a computer, a plurality ofcalibration-use images; extracting, by the computer, from each of theplurality of calibration-use images, at least grating points indicatinga distortion in a projected image of one of the plurality of projectionunits and alignment points of the projected image of the one of theplurality of projection units or a projected image of another one of theplurality of projection units; converting, by the computer, onto acommon coordinate system, the grating points of the projected images ofthe plurality of projection units extracted from the plurality ofcalibration-use images at the extracting, based on alignment pointscommon to the plurality of calibration-use images; calculating, by thecomputer, a geometric correction coefficient for providing a projectionimage to be projected from the plurality of projection units, based onthe grating points on the common coordinate system converted at theconverting; projecting, by the plurality of projection units before thepreparing the plurality of calibration-use images, a calibration imageincluding both of or one of a grating pattern defining grating points ofthe projected image and an alignment pattern defining alignment pointsbetween the plurality of calibration-use images; and preparing, beforethe projecting, a plurality of calibration projection scenes, such that(a) in an arrangement of the projected images, projection units adjacentto each other among the plurality of projection units do not bothproject a grating pattern in the same calibration projection scene, (b)all of the plurality of calibration projection scenes include at leastone grating pattern of the plurality of projection units, and (c) theplurality of calibration projection scenes constitute a tree structureby a connection based on an alignment pattern projected by one of theplurality of projection units common to the plurality of calibrationprojection scenes, wherein the preparing the plurality ofcalibration-use images includes receiving input of the plurality ofcalibration-use images including a calibration-use projected imageprojected by at least one of the plurality of projection units.
 11. Theprojection method according to claim 10, further comprising: detecting,by the computer, for each of the plurality of projection units, anoverlapping area between a projected image of a target projection unitand a projected image of an adjacent projection unit adjacent to thetarget projection unit; calculating, by the computer, a blendingcoefficient for adjusting an overlapping amount between the projectedimage of the target projection unit and the projected image of theadjacent projection unit; generating, by the computer, an intermediateimage for each of the plurality of projection units from an image to beprojected, based on the calculated geometric correction coefficient;calculating, by the computer, the projection image for each of theplurality of projection units from the intermediate image, based on theblending coefficient calculated at the blending; and projecting, by eachof the plurality of projection units, the calculated projection imagefor each of the plurality of projection units.